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ABSTRACT 

One  of  the  challenges  of  imaging  satellites  in  the  daytime  is  that  the  sun  is  generally  behind  the  satellite  from  the 
observer’s  point  of  view.  This  means  that  much  of  the  satellite  structure  can  be  in  shadow  at  any  given  time.  The 
Air  Force  Maui  Optical  and  Supercomputing  (AMOS)  observatory’s  3.6  meter  telescope  has  the  capability  of 
recording  data  simultaneously  in  two  bands  of  long-wave  infrared  (LW1R)  as  well  as  visible.  This  presents  the 
possibility  of  performing  joint  processing  of  the  infrared  and  visible  imagery,  which  is  appealing  because  the 
thermal  imagery  will  not  have  any  shadows.  We  describe  exploitation  strategies  for  this  type  of  data,  show  the 
results  of  joint  processing,  and  compare  with  single-band  images. 

1.0  INTRODUCTION 

The  3.6  meter  Advanced  Electro-Optical  System  (AEOS)  telescope  at  the  AMOS  observatory  is  a  satellite-tracking 
telescope  that  is  capable  of  simultaneously  recording  data  with  sensors  that  are  sensitive  to  visible  and  long-wave 
infrared  (LWIR)  wavelengths.  Visible  satellite  images  taken  during  daytime  conditions  are  processed  into  high- 
resolution  images  with  a  variety  of  techniques  [1,2],  but  satellite  characteristics  and  pass  geometry  can  still  result  in 
details  that  are  difficult  to  see  even  if  the  resolved  images  have  resolution  that  is  close  to  diffraction  limited.  LWIR 
images  suffer  less  from  shadowing  but  also  have  much  worse  resolution  than  visible  images.  Combining 
information  from  both  wavebands  should  allow  for  the  creation  of  high-resolution  images  with  details  that  are  easier 
to  observe.  Furthermore,  there’s  the  possibility  that  fuzzy  edge  detection  techniques  could  be  used  with  LWIR 
images  to  create  an  object  support  constraint  that  would  further  improve  the  initial  image  processing  results. 

In  section  2,  the  AEOS  LWIR  sensor  and  recent  developments  in  LWIR  data  processing  are  described.  Section  3 
shows  results  from  techniques  that  use  LWIR  images  to  improve  the  quality  of  well-resolved  visible  images, 
enhancing  dark  regions  and  reducing  remaining  blur.  Section  4  shows  results  from  using  LWIR  imagery  as  an 
object  support  constraint  in  blind  deconvolution  processing  of  visible  imagery.  Our  conclusions  are  summarized  in 
section  5. 

2.0  AEOS  LWIR  SENSOR 

The  AEOS  LWIR  sensor  is  designed  to  provide  imagery  of  targets  in  long  wave  infrared  wavebands.  A  dichroic 
mirror  splits  the  light  beam  onto  two  focal  plane  arrays  so  that  data  in  two  different  infrared  wavebands  may  be 
recorded  simultaneously.  Data  analysts  at  the  Maui  High  Performance  Computing  Center  (MHPCC)  use  the  images 
to  perform  luminosity  and  temperature  measurements. 

Recent  updates  to  the  LWIR  system  include  automation  of  processing  and  optimization  of  processing  algorithms. 
Images  are  now  automatically  transferred  from  data  acquisition  systems  to  data  processing  systems.  Radiometric 
calibration  star  collections  undergo  automated  data  quality  assessments.  Good  star  data  is  used  to  calibrate 
luminosity  measurements  of  the  target.  Temperature  measurements,  luminosity  curves,  and  collection  metadata  are 
included  in  an  automatically  generated  report  that  is  then  sent  to  interested  parties.  Fig.  1  shows  a  diagram 
describing  how  LWIR  data  is  processed. 
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LWIR  Processing 

^Radiometric/ 
Temperature, 
Geometric 
^calculations  J 

Report 

Generation 

RAW  Data  — *  Calibrate  — ► 

Fig.l.  Flowchart  of  the  LWIR  processing 

3.0  IMAGE  DESHADOWING 

Data  was  simultaneously  recorded  on  the  visible  imaging  sensor  with  an  1-band  filter  and  on  the  LWIR  sensor 
during  high  elevation  passes  of  the  Fermi  Gamma -ray  Space  Telescope,  formerly  the  Gamma-ray  Large  Area 
Telescope  (GLAST),  and  the  Hubble  Space  Telescope  (HST).  The  visible  data  was  processed  with  a  multi-frame 
blind  deconvolution  (MFBD)  algorithm  [1],  successfully  removing  most  of  the  atmospheric  blur.  A  sample  ofthe 
MFBD  processed  images  and  the  calibrated  LWIR  images  in  the  same  field  of  view  (FOV)  are  shown  in  Fig.  2. 


(a) 


(b) 


(c)  (d) 

Fig.  2.  (a)  Visible  GLAST  data  after  MFBD  processing,  (b)  visible  HST  data 
after  MFBD  processing,  (c)  LWIR  GLAST  and  (d)  LWIR  HST  data 

The  LWIR  images  are  significantly  blunder  than  the  MFBD-processed  visible  images,  but  they  also  do  not  suffer 
from  shadowing  effects  that  are  present  in  the  visible  images.  In  particular,  the  LWIR  images  show  the  presence  of 
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GLAST’s  solar  panels  and  HST’s  solar  panels  and  antennae,  but  these  features  in  the  visible  imagery  are  difficult  to 
see,  especially  HST’s  antennae.  The  dim  features  in  the  visible  imagery  can  be  brightened  with  a  simple  grayscale 
contrast  enhancement,  but  this  often  results  in  the  illumination  of  hazy  artifacts  that  are  left  by  the  MFBD  algorithm, 
as  shown  in  Fig.  3. 

Several  image  enhancement  techniques  have  been  developed,  implemented,  and  tested  on  these  passes.  One  of  the 
most  successful  methods  involves  using  the  inverse  of  the  LWIR  image  as  a  partially  opaque  mask  that  non- 
uniformly  decreases  the  values  of  pixels  in  a  combined  visible-LWIR  image.  The  visible  and  LWIR  images  are 
normalized  to  unity,  and  then  the  LWIR  image  is  registered  to  the  visible  image.  Each  pixel  of  a  combined  image  is 
calculated  by  taking  the  mean  value  of  the  LWIR  and  visible  pixels  at  that  location.  This  step  is  necessary  in  order 
to  include  features  that  are  missing  in  the  LWIR  images,  such  as  the  hatch  on  HST.  The  result  is  inverted  and  shifted 
upward  by  a  small  scalar  value  in  order  to  not  have  any  zero-value  elements  in  the  final  mask.  The  partial-opacity 
masks  that  result  from  this  process  are  shown  in  Fig.  4. 


(a)  (b) 

Fig.  3.  (a)  Visible  GLAST  MFBD  data  and  (b)  visible  HST  MFBD  data  after  contrast  enhancement 


(a)  (b) 


Fig.  4.  (a)  GLAST  and  (b)  HST  partial-opacity  masks 
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(a)  (b) 

Fig.  5.  Visible  (a)  GLAST  and  (b)  HST  data  after  the  full 
partial-opacity  masking  procedure  has  been  applied 

The  MFBD-processed  visible  images  shown  in  Fig.  2  are  multiplied  element-wise  by  the  inversion  masks  in  Fig.  4 
to  produce  the  enhanced  images  in  Fig.  5.  The  panels  and  antennae  are  more  visible  in  the  resulting  image,  but  there 
is  still  a  small  amount  of  haze  obfuscating  the  panel  edges.  This  is  much  more  prevalent  in  the  Flubble  pass  where 
the  panels  were  already  close  to  the  background  values.  These  results  are  promising;  the  dimmer  features  of  the 
object  become  easier  to  see,  and  overall  there  is  less  haze  than  would  appear  due  to  a  normal  contrast  adjustment. 

The  partial-opacity  enhancement  technique  can  be  improved  upon  by  using  a  Laplacian  pyramid  [3]  to  break  down 
the  images  by  frequency  values  before  generating  and  applying  the  mask.  Both  the  visible  and  LW1R  images  are 
broken  down  with  a  Laplacian  pyramid,  and  then  the  low  frequency  components  are  enhanced  with  the  partial 
opacity  mask.  The  resulting  images  are  then  recombined  to  produce  the  images  in  Fig.  6.  The  dynamic  range  has 
been  significantly  reduced,  allowing  the  dimmer  aspects  of  the  image  to  become  visible  but  without  the  hazy 
artifacts  present  in  Fig.  5.  An  additional  contract  enhancement,  shown  in  Fig.  7,  demonstrates  that  most  of  the  haze 
around  the  object  has  been  completely  removed  while  leaving  behind  the  desirable  features. 


(a)  (b) 

Fig.  6.  Visible  (a)  GLAST  and  (b)  FIST  MFBD-processed  images 
after  Laplacian  pyramid  partial-opacity  masking  enhancement. 
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(a)  (b) 

Fig.  7.  Visible  (a)  GLAST  and  (b)  HST  MFBD-processed  images  after  Laplacian  pyramid 
partial-opacity  masking  enhancement  and  additional  gamma  contrast  enhancement. 

4.0  MULTI-FRAME  BLIND  DECONVOLUTION  WITH  LWIR  MASKING 

Multi-frame  blind  deconvolution  (MFBD)  is  a  technique  for  deconvolving  a  pristine  image  from  a  blurring  function. 
Our  blurring  function  is  a  point  spread  function  (PSF)  dominated  by  atmospheric  turbulence.  Short-exposure 
images  are  used  to  estimate  the  PSF  and  produce  an  image  with  resolution  approaching  the  diffraction  limit  of  the 
optical  system.  Part  of  the  MFBD  process  includes  defining  a  region  of  the  image  for  an  initial  estimate  of  the 
object,  called  the  object  support  region.  It  can  be  difficult  to  algorithmically  calculate  a  good  object  support  region 
if  the  atmospheric  blur  is  particularly  strong.  Images  recorded  in  IR  wavebands  are  less  susceptible  to  turbulence 
effects  and  can  produce  a  rough  estimate  of  an  object  support  constraint  for  use  with  MFBD  processing. 

Converting  an  LWIR  image  into  an  object  support  constraint  requires  converting  the  pixels  that  contain  object 
information  into  a  mask.  The  use  of  long  wavelengths  results  in  blurry  images  with  edges  that  are  difficult  to  detect 
with  algorithms  like  Canny  edge  detection  [4].  A  pixel  threshold  filter  can  produce  an  acceptable  mask,  but  with  the 
risk  of  losing  faint  details.  The  Fast  Multilevel  Fuzzy  Edge  Detection  (FMFED)  algorithm  [5]  provides  a  good 
estimate  of  the  object  shape.  This  algorithm  produces  a  conservative  outline  of  the  object,  preserving  faint  details 
while  suppressing  noise  information.  An  example  of  an  LWIR  image  being  converted  into  a  mask  with  the  FMFED 
algorithm  is  shown  in  Fig.  8.  LWIR  images  are  recorded  in  a  much  larger  field  of  view  than  visible  images,  so  the 
mask  has  to  be  expanded  and  cropped  to  the  same  dimensions  as  the  visible  imagery.  The  expanded  mask  and  the 
expanded  mask  overlaid  with  an  expanded  LWIR  image,  for  demonstrative  purposes,  are  shown  in  Fig.  9. 


Hubble  Space  Telescope  -  Raw  LWIR 


Hubble  Space  Telescope  FMFED  edges 


Hubble  Space  Telescope  -  Mask 


(a)  (b)  (c) 

Fig.  8.  (a)  Raw  LWIR  image  of  HST,  (b)  FMFED  edges,  and  (c)  FMFED  mask 
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Hubble  Space  Telescope  -  LWIR  mask 

n 

(a)  (b) 

Fig.  9.  (a)  The  mask  rescaled  and  cropped  to  same  FOV  as  visible  images  and 
(b)  the  rescaled  mask  on  top  of  the  rescaled  LWIR  image 

The  MFBD  algorithm  was  used  to  process  the  raw  visible  images  with  the  mask  from  Fig.  9a  provided  as  an  object 
support  constraint.  For  comparison,  the  MFBD  algorithm  was  run  a  second  time  without  an  object  support 
constraint.  The  results  are  shown  in  Fig.  10  and  Fig.  11,  but  they  are  nearly  identical  to  each  other.  A  line  spread 
function  extraction  technique  [5]  was  chosen  as  a  suitable  method  for  assessing  the  relative  quality  of  these  images, 
but  it  indicated  approximately  no  difference  in  quality  regardless  of  whether  the  LWIR  mask  was  provided.  Image 
erosion  and  dilation  algorithms  were  applied  to  the  masks  in  order  to  tighten  or  loosen  the  constraint  size,  but  there 
was  still  no  improvement  in  the  quality  of  the  resultant  images. 

Hubble  Space  Telescope  -  MFBD  results  with  no  mask 


\ 


Raw  Data  40  Iterations  80  Iterations 


120  Iterations  160  Iterations  200  Iterations 

Fig.  10.  MFBD  results  without  a  provided  object  support  constraint. 
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Hubble  Space  Telescope  -  MFBD  results  with  LWIR  support 


Raw  Data  40  Iterations  80  Iterations 


120  Iterations  160  Iterations  200  Iterations 

Fig.  11.  MFBD  results  with  an  LWIR  mask  provided  as  the  object  support  constraint 
and  with  the  LWIR  mask  overlaid  on  top  of  the  results  for  illustrative  purposes 


5.0  CONCLUSIONS 


The  AEOS  telescope’s  LWIR  sensor  has  been  shown  to  provide  image  quality  enhancements  of  simultaneously 
recorded  visible  images.  The  best  results  were  obtained  by  using  a  Laplacian  pyramid  to  split  the  LWIR  and 
reconstructed  visible  images  into  several  frequency  ranges,  using  the  LWIR  images  to  create  a  partial-opacity  mask 
that  darkens  the  brightest  parts  of  the  visible  satellite  images  in  specific  frequency  ranges,  and  then  recombining  the 
visible  images  into  a  final  enhanced  image.  We  were  successfully  able  to  produce  cleaner  images  with  fewer 
shadows  than  the  unenhanced  MFBD  results. 

Attempts  were  made  to  use  the  LWIR  images  as  an  object  support  mask  for  MFBD  processing,  but  no  improvement 
was  observed.  It  is  possible  that  LWIR  support  in  MFBD  may  only  be  beneficial  when  the  atmospheric  PSF  has  a 
much  larger  angular  extent  than  the  LWIR  measurement.  Future  work  will  include  an  expanded  survey  of 
simultaneous  passes  for  LWIR-mask  MFBD  processing  in  varied  turbulence  conditions. 
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